Accurately predicting fluid flow through fractured media remains a major challenge due to the disparity between simple modelling assumptions and the complex reality of fracture geometry. Fractures at the continuum-scale are represented as very simple geometries, but in reality, they are quite complex. Assuming a parallel plate-like geometry in our numerical models can yield very high errors for fluid transport. While there have been numerous attempts to come up with a relationship that conveys micro-scale information about how fracture geometry influences flow at the continuum-scale, a universal equation remains elusive. This is partly due to the fact that effective properties used in these relationships fail to capture enough of the complexity of real fracture geometries. Machine learning approaches are promising, but integrating flow physics as hard constraints in architectures has not been possible. Here, we introduce a novel application of differentiable programming in geosciences, enabling data-driven learning that adheres to fundamental conservation laws. In this work, we reproduced fractures geometries in silico by the spectral method implemented in the Python library pySimFrac. Then we created a dataset of three-dimensional resolved single-phase flow in fractures through LBM simulations, this is the ground truth for our trainings. Then, we trained a convolutional neural network to predict the permeability field, which is the input to a differentiable macroscale solver that provides the velocity field in that fracture. The loss evaluation is based on the velocity output from the macroscale code to ensure it matches the LBM ground truth. We demonstrate that our approach generalizes well to unseen fracture geometries, outperforming the local cubic law in predicting flow characteristics. By using high-resolution lattice Boltzmann simulations to train our neural network, we ensure accurate permeability field predictions for new fracture configurations, leading to improved velocity field estimates when used with a continuum-scale differentiable solver. This novel approach connects micro- and continuum-scale behaviour, allowing us to come up with a general model for the permeability of rough, complex fractures. Our work paves the way for significantly improved flow predictions and ultimately a deeper understanding of multi-scale flow dynamics.